In recent years, the number of pixels of an imaging device used in an image inputting apparatus has been remarkably increased. The resolution of the imaging device has been improved to the level of a smooth granularity that human eyes cannot recognize. Under such circumstances, requirements for picture quality have been changed from higher resolution to higher color reproducibility, higher noise reduction, and wider dynamic range. The present invention provides a means and an apparatus that can increase a luminance dynamic range they scan while maintaining picture quality, in particular, color reproducibility and noise characteristic in their allowable levels.
An image processing apparatus that can more faithfully reproduce colors and reduce noise than before is disclosed in Patent Document 1 (Japanese Patent Application Unexamined Publication No. 2003-284084).
Next, the apparatus disclosed in Patent Document 1 will be described. A four-color filter 1 having an array of colors shown in FIG. 1 is disposed on the front of an imaging device. As denoted by dotted lines, the color filter 1 has a structure of four-color units each of which is the minimum unit composed of an R filter that transmits only red (R) light, a B filter that transmits only blue (B) light, a G1 filter that transmits only green light of a first wavelength band, and a G2 filter that transmits only green light of a second wavelength band.
FIG. 2A and FIG. 2B show an example of a structure of a signal processing section that performs a signal process for an image signal captured by an imaging device for example a CCD which has the color filter 1. Because of the limited drawing space, the structure of the signal processing section is shown in two portions of FIG. 2A and FIG. 2B. Reference numeral 10 denotes a frontend to which four types of color signals (R signal, G1 signal, G2 signal, and B signal) are input from an image sensor. The frontend 10 performs a correlated double sampling process, a gain controlling process, a digital transforming process, and so forth for the color signals supplied from the image sensor. The correlated double sampling process is performed to remove noise components which are output from the color signals. Image data which are output from the frontend 10 are supplied to a signal processing section 11 composed of an LSI (Large Scale Integrated Circuit).
The signal processing section 11 is connected to a microcomputer (not shown) through a microcomputer interface 12. The microcomputer controls the whole operations of for example a digital still camera according to a predetermined program. In addition, the microcomputer controls each block that composes the signal processing section 11 through the microcomputer interface 12.
The signal processing section 11 performs an interpolating process, a filtering process, a matrix calculating process, a luminance signal generating process, a color difference signal generating process, and so forth for four types of color signals which are input from the frontend 10. The frontend 10 generates a picture signal. The picture signal is supplied to a display unit (not shown). The display unit displays the imaged picture. In addition, picture data which are output from the signal processing section 11 are compressed and stored to an internal storage medium, an external storing medium, or the like.
Next, each block of the signal processing section 11 will be described. An offset compensation processing section 21 removes a noise component (offset component) from the picture signal supplied from the frontend 10. The picture signal supplied from the offset compensation processing section 21 is output to a white balance compensation processing section 22. The white balance compensation processing section 22 compensates a white balance of the picture signal. In other words, the white balance compensation processing section 22 compensates unbalanced colors due to the difference of color temperatures of an object and the difference of the sensitivities of the color filters (R, G1, G2, and B) of the sensor.
An output of the white balance compensation processing section 22 is supplied to a vertical direction synchronization processing section 23. The vertical direction synchronization processing section 23 synchronizes picture data that chronologically differ in the vertical direction using a delaying device for example a small capacity memory so as to perform a vertical direction interpolating process and a vertical direction filtering process for the picture data.
A plurality of picture signals synchronized by the vertical direction synchronization processing section 23 are supplied to a processing section 24 for an interpolating process, a filtering process, a high frequency compensating process, and a noise process.
The processing section 24 performs an interpolating process for interpolating color signals of 2×2 pixels, which is the minimum unit of the color filters (R, G1, G2, B), with phases in the same space, a filtering process for properly limiting a signal bandwidth, a high frequency compensating process for compensating a high frequency component of the signal bandwidth, a noise process for removing noise components from the signals, and other processes.
Picture signals for example signals of four colors of RG1G2B obtained by the processing section 24 are supplied to a linear matrix processing section 25. The linear matrix processing section 25 performs matrix calculations for four inputs and three outputs. When matrix coefficients of a 3×4 matrix are given, RGB color outputs can be obtained from picture information of four colors of RG1G2B.
RGB outputs of the linear matrix processing section 25 are supplied to gamma compensation processing sections 26R, 26G, and 26B, respectively. The gamma compensation processing sections 26R, 26G, and 26B inversely compensate nonlinear characteristics of the display unit and finally accomplish linear characteristics.
Output signals of the gamma compensation processing sections 26R, 26G, and 26B are supplied to each of a luminance (Y) signal generation processing section 27 and a color difference (C) signal generation processing section 28. The luminance (Y) signal generation processing section 27 combines gamma-compensated RGB signals with predetermined combining ratios and generates a luminance signal. The color difference signal generation processing section 28 combines the gamma-compensated RGB signals with predetermined combining ratios and generates a color different signal.
The color difference signal generated by the color difference signal generation processing section 28 is supplied to a bandwidth limitation and thin-out processing section 29. The bandwidth limitation and thin-out processing section 29 generates a color difference signal of which color difference signals Cb and Cr have been time-division multiplexed. Thus, it can be said that the image processing apparatus using the four-color filter is superior to that using the three-primary-color filter in color reproducibility.
Generally, it is preferred that as spectral sensitivity the imaging device have high color reproducibility and good noise characteristic. “Good color reproducibility” means that the imaging device can sense the same colors as human eyes can or the difference between colors that the imaging device can sense and the human eyes can see is small. What the human eyes can see means that colors that they can see. “Good noise characteristic” means that the noise amount in a particular luminance level is small. Noise is largely categorized as luminance noise and color noise. The luminance noise depends on the absolute sensitivity of the imaging device. In contrast, the color noise largely depends on the relationship between spectrum sensitivities of color filters of the imaging device, namely the shape of a spectral sensitivity curve.
A method of matrix-transforming output signals of an imaging device which are linearly proportional to the luminance and generating primary color RGB signal values is often used in a signal process of a conventional image inputting apparatus shown in FIG. 2A and FIG. 2B. This process is referred to as the linear matrix process. In many cases, since an input image of an image inputting apparatus (for example, a scanner or a digital camera) is observed and edited on a monitor of a personal computer (hereinafter sometimes referred to as the PC), the target color space of primary color RGB signal values that have been matrix-calculated is designated to an sRGB color space, which is the color space of the PC monitor.
IEC (International Electrotechnical Commission) has defined the sRGB color space as a standard multimedia color space on the basis that color image signals should be transmitted. When the transmission side transmits a color image and the reception side receives it on the basis of the standard color space, they can share the same color reproduction.
Thus, the target spectral sensitivity (referred to as the relative sensitivity in the drawings) of the imaging device is an sRGB color matching function of which a color matching function (refer to FIG. 3), which is the spectral sensitivity of human eyes, is linearly transformed by a 709 type matrix. For details of the 709 type matrix, refer to Reference Document 1 (“ITU-R BT. 709-3, “Basic Parameter Values for the HDTV standard for the Studio and for International Programme Exchange” (1998)”).
In FIG. 3, a curve 31x denotes a function x(λ), a curve 31y denotes a function y(λ), and a curve 31z denotes a function z(λ). The graph of the color matching function shown in FIG. 3 is defined as the CIE (Commission International de l'Eclairage) 1931.
FIG. 4 is a graph showing the sRGB color matching function. In FIG. 4, a curve 32r denotes a function r(λ), a curve 32g denotes a function g(λ), and a curve 32b denotes a function b(λ). Since the sRGB color matching function satisfies a router condition, the imaging device can sense colors that the human eyes can see. For details of the router condition, refer to Reference Document 2 (Noboru Ohta, “Engineering on Chromatics (translated title)”, ISBN 4-501-61350-5, Tokyo Denki-Daigaku Publishing Office (1993)).
However, since the spectral sensitivity shown in FIG. 4 contains a negative component, it is practically impossible to produce a three-color RGB filter having such spectral sensitivity. If a three-color RGB filter that has positive spectral sensitivity and satisfies the router condition, the filter has spectral sensitivity as shown in FIG. 5. In FIG. 5, a curve 33R denotes a function sR(λ), a curve 33G denotes a function sG(λ), and a curve 33B denotes an sB(λ).
As is clear from FIG. 5, the curve 33R of the spectral sensitivity of the red component filter of the imaging device largely matches the curve 33G of the spectral sensitivity of the green component filter. This means that two component signals are very similar. Thus, when three colors of the sRGB space as target output signals of the imaging device using the filters having the spectral sensitivities shown in FIG. 5 are calculated, matrix calculations given by formula (1) are required.
                              (                                                                                          r                    _                                    ⁡                                      (                    λ                    )                                                                                                                                            g                    _                                    ⁡                                      (                    λ                    )                                                                                                                                            b                    _                                    ⁡                                      (                    λ                    )                                                                                )                =                              (                                                            6.5614                                                                      -                    5.5412                                                                    0.1845                                                                                                  -                    2.0049                                                                    3.1163                                                                      -                    0.1635                                                                                                0.1182                                                                      -                    0.2783                                                                    1.0688                                                      )                    ·                      (                                                                                                      S                      R                                        ⁡                                          (                      λ                      )                                                                                                                                                              S                      G                                        ⁡                                          (                      λ                      )                                                                                                                                                              S                      B                                        ⁡                                          (                      λ                      )                                                                                            )                                              (        1        )            
As is clear from the matrix coefficients of formula (1), to calculate the red component of the output signals, the red component and the green component of the input signals are multiplied by very large matrix coefficients of 6.5614 and −5.5412. This means that noise of the red signal and the green signal of the imaging device is much increased.
Thus, a three-color RGB filter that does not perfectly satisfy the router condition, namely that has good noise characteristic with sacrifice of color reproducibility to some extent, but that has spectral sensitivity as shown in FIG. 6 is used. In FIG. 6, a curve 34R denotes a function s1R(λ), a curve 34G denotes a function s1(G), and a curve 34B denotes a function s1B(λ). Since the spectral sensitivity shown in FIG. 6 does not satisfy the router condition, the spectral sensitivity of the three-color filter shown in FIG. 6 cannot be linearly transformed into the sRGB color matching function. Thus, approximately transformed matrix calculations for the sRGB color matching function is given by formula (2).
                              (                                                                                          r                    ^                                    ⁡                                      (                    λ                    )                                                                                                                                            g                    ^                                    ⁡                                      (                    λ                    )                                                                                                                                            b                    ^                                    ⁡                                      (                    λ                    )                                                                                )                =                              (                                                            2.250                                                                      -                    0.649                                                                                        -                    0.089                                                                                                                    -                    0.057                                                                    1.574                                                                      -                    0.384                                                                                                                    -                    0.009                                                                                        -                    0.444                                                                    1.567                                                      )                    ·                      (                                                                                s                    ⁢                                                                                  ⁢                                          1                      R                                        ⁢                                          (                      λ                      )                                                                                                                                        s                    ⁢                                                                                  ⁢                                          1                      G                                        ⁢                                          (                      λ                      )                                                                                                                                        s                    ⁢                                                                                  ⁢                                          1                      B                                        ⁢                                          (                      λ                      )                                                                                            )                                              (        2        )            
The absolute values of all the matrix coefficients given by formula (2) are smaller than those given by formula (1). Thus, it is clear that when colors are separated, noise is not relatively increased.
Because of the foregoing reason, it is known that the typical primary color system RGB imaging device has spectral sensitivity denoted by a curve as shown in FIG. 6 and has excellent color reproducibility and noise characteristic. However, the imaging device actually has spectral sensitivity denoted by a curve as shown in FIG. 7 because of the influences of the sensitivity of the imaging device itself, the characteristics of a lens, and the characteristics of an infrared cut filter. In FIG. 7, a curve 35R denotes spectral sensitivity of an R filter, a curve 35G denotes spectral sensitivity of a G filter, and a curve 35B denotes spectral sensitivity of a B filter.
Due to various restrictions in production of color filters, it is difficult to improve their spectral sensitivities. To maintain their spectral sensitivities, for example the cell size may be increased or an electric gain may be applied. However, if the cell size is increased, improvement of resolution is sacrificed. If the gain is applied, noise reduction is sacrificed.
As a result, although the primary color system RGB imaging device has excellent color reproducibility, it can be more improved. In addition, although the primary color system RGB imaging device has low color separation noise, luminance noise as a dominant component of noise tends to increase due to low sensitivity. In other words, although the primary color system RGB imaging device has excellent color reproducibility, it tends to have large noise.
A complementary color system color filter is also known as a color filter. An complementarily checkered line sequence type color filter of which four color filters of for example Y (yellow), C (cyan), M (magenta), and G (green) are arrayed as shown in FIG. 8 is known. In FIG. 8, a matrix of 2×4 denoted by dotted lines is the minimum unit.
FIG. 9 is a graph showing spectral sensitivity of the complementary color system YCMG imaging device. In FIG. 9, a curve 36Y denotes spectral sensitivity of a Y filter, a curve 36C denotes spectral sensitivity of a C filter, a curve 36M denotes spectral sensitivity of an M filter, and a curve 36G denotes spectral sensitivity of a G filter. As shown in FIG. 9, since the sensitivity of each color filter is high, the complementary color system YCMG imaging device advantageously images an object at a dirk place and has excellent luminance noise characteristic. However, since spectral sensitivities of the color filters largely overlap, if the color reproducibility of the imaging device is improved, very large color separation coefficients are required. As a result, color separation noise increases. Thus, the color reproducibility of the complementary color system YCMG imaging device cannot be improved in comparison with that of the primary color system RGB imaging device.
Thus, there is much room for improving the color reproducibility of the complementary color system YCMG imaging device. In contrast, the complementary color system YCMG imaging device has large color separation noise. However, since the sensitivity of the complementary color system YCMG imaging device is high, luminance noise which is a dominant component of noise of the complementary color system YCMG imaging device tends to decrease. In other words, the complementary color system YCMG imaging device has sufficient noise characteristic and insufficient color reproducibility.
As described above, when the conventional primary color system RGB imaging device is used, luminance noise becomes large due to low sensitivity. In contrast, when the conventional complementary color system YCMG imaging device is used, color reproducibility and color separation noise characteristic are insufficient.
Therefore, an object of the present invention is to provide an imaging apparatus and an imaging device that can solve the problems of the forgoing imaging devices.